Skip to content

Stop the plugin wildcard breaking the Zabbix agent install - #5

Open
MarkLFT wants to merge 3 commits into
mainfrom
zabbix-plugin-install-fix
Open

Stop the plugin wildcard breaking the Zabbix agent install#5
MarkLFT wants to merge 3 commits into
mainfrom
zabbix-plugin-install-fix

Conversation

@MarkLFT

@MarkLFT MarkLFT commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Installing the Zabbix agent on a fresh Debian 13 host failed mid-apt, leaving zabbix-agent2 half-configured and aborting the script.

Root cause — proved, not guessed

The install line used the plugin wildcard:

apt-get install ... zabbix-agent2 zabbix-agent2-plugin-*

That pulls in zabbix-agent2-plugin-nvidia-gpu. On a host with no NVIDIA driver, the agent's own config test aborts:

ERROR: Cannot register plugins: failed to register metrics of plugin "NVIDIA":
failed to create new nvml runner: NVML error: NVML Shared Library couldn't be found or loaded.

A/B in a clean Debian 13 container:

Installed zabbix_agent2 -T
zabbix-agent2 alone Validation successful, exit 0
zabbix-agent2 + the wildcard exit 1, NVML error above

zabbix-agent2.service has ExecStartPre=/usr/sbin/zabbix_agent2 -T -c $CONFFILE, so the failing config test fails the service start, which fails dpkg's postinst, which leaves the package unpacked but never configured — and every later apt call complains until that is cleared.

The script already has a "disable unneeded loadable plugins" step written for exactly this failure (the comment even names the NVIDIA plugin). But it runs after the install, and dpkg starts the agent during the install — so on a fresh install the guard could never fire. It only ever protected a re-run.

Changes

  1. Install zabbix-agent2 only. Loadable plugins are installed per detected service; postgresql now joins the existing mssql and redis handling, so the wildcard's useful coverage is preserved without installing plugins for hardware and services the host doesn't have.
  2. Defer service starts during package work with a temporary policy-rc.d, so no plugin can fail the package configure again. Restored before the agent is started, with an EXIT trap so it's never left behind.
  3. Clear a pre-existing half-configured dpkg state before installing. With starts deferred this succeeds where it previously failed — so re-running the script recovers a host already stuck in this state, which is the situation the reported failure leaves behind.
  4. Validate config before starting, so a failure reports the agent's own message instead of a bare systemd exit code.

A bug I introduced and caught

My first version's EXIT trap called rm -f /usr/sbin/policy-rc.d unconditionally — on any run that never deferred (e.g. the reconfigure path), that would have deleted a policy-rc.d belonging to the host, silently changing how every future package install behaves. It now tracks whether it created the file and only removes its own; a pre-existing one is set aside and restored. Test 1 below covers exactly this.

Test results — Debian 13 containers, 10/10

1. policy-rc.d semantics

  • pre-existing policy-rc.d untouched when we never deferred
  • defer installs ours (exit 101)
  • original restored afterwards, byte-identical

2. Fresh install (the fix)

  • apt install exits 0
  • dpkg --audit clean — nothing half-configured
  • zabbix_agent2 -T passes
  • nvidia-gpu plugin not installed (no plugin packages at all on a bare host)

3. Recovery of a host broken exactly as reported

  • reproduced the failure by installing the nvidia plugin with starts allowed
  • dpkg --configure -a reconciles with starts deferred
  • config test passes once the plugin's config is disabled

bash -n and shellcheck 0.10.0 clean.

Operator note

To unblock an already-broken host without waiting for this to merge:

sudo apt-get purge -y zabbix-agent2-plugin-nvidia-gpu
sudo dpkg --configure -a

After this merges, simply re-running the script does the same thing.

Mark added 3 commits August 2, 2026 12:11
Installing the agent on a fresh Debian 13 host failed mid-apt, leaving
zabbix-agent2 half-configured and aborting the script.

Root cause, proved by A/B in a container: the install line used the
zabbix-agent2-plugin-* wildcard, which pulls in zabbix-agent2-plugin-nvidia-gpu.
On a host with no NVIDIA driver the agent's own config test aborts with "NVML
Shared Library couldn't be found or loaded". Agent alone validates successfully;
agent plus the wildcard exits 1. Because ExecStartPre runs that config test, the
service fails to start, the postinst fails with it, and dpkg is left with the
package unpacked but never configured.

The script already has a "disable unneeded loadable plugins" step written for
exactly this failure, but it runs after the install, so on a fresh install it
can never fire — it only ever protected a re-run.

Changes:

- Install zabbix-agent2 only. Loadable plugins are installed per detected
  service; postgresql now joins the existing mssql and redis handling, so the
  wildcard's useful coverage is preserved without installing plugins for
  hardware and services the host does not have.
- Defer service starts for the duration of the package work with a temporary
  policy-rc.d, so no plugin can fail the package configure again. It is restored
  before the agent is started, with an EXIT trap so it is never left behind, and
  it only ever removes the file it created — a pre-existing policy-rc.d belongs
  to the host and now survives.
- Clear a pre-existing half-configured dpkg state before installing. With starts
  deferred this succeeds where it previously failed, so re-running the script
  recovers a host already stuck in this state.
- Validate the config before starting the service, so a failure reports the
  agent's own message instead of a bare systemd exit code.

Tested 10/10 in Debian 13 containers: policy-rc.d save/restore semantics, fresh
install (clean dpkg state, config test passes, nvidia plugin not installed),
reproduction of the reported failure, and recovery of a host broken by it.
Zabbix's Debian/Ubuntu packages carry an epoch, e.g. 1:7.4.13-1+debian13.
Comparing that raw against the target major.minor ('7.4') never matched, so the
'already on target version, nothing to do' short-circuit could never fire and
every run — including every scheduled TacticalRMM run — re-entered the full
repository-add and install path.

Strip the epoch before comparing, and anchor on the dot so 7.4 does not match
7.44. Verified against real package version strings.
Progress.md is updated once, on the repair-script branch. Keeping the same
edits on three branches would conflict on every merge after the first.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant